DOCUMENT RESUME 

ED 351 517 CE 062 317 



TITLE Criterion-Referenced Test Items for Vocational 

Education* Final Report* 
INSTITUTION Missouri Univ., Columbia. Instructional Materials 

Lab. 

SPONS AGENCY Missouri State Dept. of Elementary and Secondary 
Education, Jefferson City. Div. of Vocational and 
Adult Education. 
30 Jun 91 
25p. 

Reports ~ Descriptive (141) 
MFOl/PCOl Plus Postage. 

Auto Mechanics; Child Caregivers; Child Care 
Occupations; Competence; ^^Computer Oriented Programs; 
'^Criterion Referenced Tests; Drafting; Electronics; 
*Inservice Teacher Education; Job Skills; Machine 
Tools; Secondary Education; Test Construction; *Test 
Items; Test Validity; Two Year Colleges; ^Vocational 
Education 
Missouri 



A project was conducted to develop and validate 
criterion-referenced test items for the following vocational 
education programs: child care worker, machine shop, electronics, 
drafting, and automobile mechanics. For each of the programs, the 
following activities took place: a project coordinator was assigned, 
subject-area instructors and a test construction expert were selected 
to participate in the test item development workshop, existing 
material was identified, a test item development workshop was 
conducted, tests were submitted for validation by test-construction 
experts, suggested changes were incorporated and field tested at 
various sites within Missouri, additional revisions accommodated 
field test results, and copies and compute* disks of the tests were 
prepared for use during an inservice meeting of the Missouri 
Vocational Association. The tests are suitable for use by secondary 
schools, vocational-technical schools, junior colleges, and other 
institutions of higher and special education. The materials are 
expected to be used in more than 400 Missouri school districts. 
(Appendixes, which make up more than half the document, include the 
following: agenda for the machine shop criterion-referenced test 
development workshop; information on writing test items, including 
suggestions for using the cognitive, affective, and psychomotor 
domains; a matrix for linking objectives to test items, advantages 
and disadvantages of various types of test items; rules for writing 
true-false, matching, and multiple-choice items, and performance 
checklists; and steps in writing test items and checklists for 
Vocational Instructional Management Systems competencies.) (KC) 



PUB DATE 
NOTE 
PUB TYPE 

EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 
ABSTRACT 



* >V Vc ?'c Vc ?'c ?V >V ic ?V ?V * -icki: Vt i< ?V Vc * V? * A Vc Vc Vc Vt^VVc ^VV: * Vc Vc Vc A^V?V* Vc oV** VcVc ***** ****** * V: 

Reproductions supplied by EDRS are the best that can be made * 
''^ from the original document. * 

***********^e***************V:*Vf***************************************** 



N 

GO 



FINAL REPORT 



Criterion-Referenced Test Items for Vocational Education 



(RFP 91-133-110-4) 



Harley Schlichting, Project Director 
University of Missouri-Columbia 
Columbia, MO 65211 



June 30, 1991 



K 
A 

ERIC 



Missouri Department of Elementary and Secondary Education 
Vocational Planning and Evaluation Unit 
Jefferson City, MO 65102 



U S DCPAWTMENT OF CDUCATlOK 
OHtce 01 EduC«l"on«l R«»«*fCh una lmofov«meo| 

EDUCATIONAL RESOURCES INFORMATION 
CENTER <ERIC) 

ttis document has been reorcxJuced as 

received from the person or Org»ni;atiOn 
Ongmatir^g «t 
' Minor Changes have been made Jo improve 

reproduction Quality 

Points of view or opmtons stated in this docu 
ment do not necessarily represent ofticiai 
OERi position or poiicy 



"PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 



2^ 



TO THE LIUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC)." 



BEST COPY AYAIIJIBIE 



ABSTRACT 

Title: Criterion-Referenced Test Items for Vocational Education (RFP 91-133-110-4) 

Principal Investigator: Hariey Schlichting, Director, Instructional Materials Laboratory 

Institution: University of Missouri-Columbia 

Location: Columbia, MO 65211 

Phone: 314/882-2884 

Starting/Ending Dates: July 1, 1990, June 30, 1991 

Objectives of the Project 

To develop and validate criterion-referenced test items for the following vocational education 
programs: 

1. Child Care Worker 

2. Machine Shop 

3. Electronics 

4. Drafting 

5. Auto Mechanics 



A Brief Description of the Project 

The number of copies prescribed in the Request for Proposals for each curriculum module will 
accompany this report to the Division of Vocational and Adult Education, Vocational Planning and Evaluation 
Section, by the scheduled completion date. 

Impact Statement 

Materials developed as a result of this project will De used by vocational educators in nearly 400 
Missouri school districts, thus reaching thousands of vocational students throughout the state. 

OBJECTIVES 

Home Economics Education - Child Care Worker 

The overall purpose of this proposal was to develop and validate one (1) test item for each 
competency/performance objective current! identified for the Missouri Child Care Worker. Phase II. 
Achieving this goal required the completion of specific objectives. 



1 . Conduct a workshop to develop one (1) test item for each performance objective, composed of six 
(6) occupational child care instructors, representatives from the Department of Elementary and 
Secondary Education, and a test construction specialist for guidance, critique, evaluation, and 
instruction in writing test items in the cognitive, affective, and psychomotor domains. Review and 
revise criterion-referenced test items. 

2. Validate test items by developing an instrument to evaluate content and construction for the writers 
and other select occupational child care instructors. 

3. Revise and finalize test items, based on evaluation critiques. 

4. Deliver elg^.t (8) copies of the final items along with (4) copies of the final report to the Vocational 
Planning and Evaluation Unit, A copy of the final product and report will be submitted to the 
Vocational Planning and Evaluation Section for review and approval before being reproduced. 

5. Provide inservice training to home economics educators/teachers through regional workshops and 
state vocational inservice conferences. 

Trade and Technical - Machine Shop 

The general objective of this proposal was to develop criterion-referenced test items for the Missouri 
machine shop competency profile. The test items could be adopted for use in either the VAMS program 
or the COMPVIMS/Project Basic III program. Successful completion of this general objective depended 
upon the completion of the following specific objectives. 

1. Assign a coordinator for the project. 

2. Select machine shop instructors and a test-construction expert to participate in the test-item 
development workshop. 

3. Identify existing material (i.e., test banks or references) that may be useful in developing the test 
items. 

4. Conduct the test-item development workshop. The three-day workshop would consist of the 
following persons: machine shop instructors; a test-construction expert; and representatives of 
DESE Industrial Education Section. 

5. Prepare the test items for validation by the test-construction expert; submit items for validation. 

6. Incorporate changes suggested by the test-construction expert. 

7. Select five field test sites, each in a different geographic location within Missouri. 

8. Orientate the field testers via teleconference. Field test the items. 

9. Conduct a meeting of field testers to evaluate items. 

10. Revise items according to the results of the field test. 



11. Deliver two copies of the completed test-item booklets and computer disks to DESE Industrial 
Education staff and three copies to the Division of Vocational and Adult Education, Vocational 
Planning and Evaluation. 

12. Provide 33 booklets and computer disks to DESE Industrial Education staff to train machine shop 
instructors in test-item use during an in-service meeting at the 1991 Missouri Vocational Association 
Summer Conference. 

Trade and Technical - Electronics 

The general objective of this proposal was to develop criterion-referenced test items for the Missouri 
electronics competency profile. The test items could be adopted for use in either the VAMS program or the 
CO MPVIMS/P reject Basic III program. Successful completion of this general objective depended upon the 
completion of the following specific objectives. 

1. Assign a coordinator for the project. 

2. Select electronics instructors and a test-oonstruction expert to participate in the test-Item 
developmr^nt workshop. 

3. Identify existing material (i.e., test banks or references) that may be useful in developing the test 
Items. 

4. Conduct the test-item development workshop. The three-day workshop would consist of the 
following persons: electronics instructors; a test-construction expert; and representatives of DESE 
Industrial Education Section. (Refer to Appendix 1) 

5. Prepare the test items for validation by the test-construction expert; submit items for validation. 

6. Incorporate changes suggested by the test-construction expert. 

7. Select five field test sites, each in a different geographic location within Missouri. 

8. Orientate the field testers via teleconference. Field test the items. 

9. Conduct a meeting of field testers to evaluate items. 

10. Revise items according to the results of the field test. 

11. Deliver two copies of the completed test-item booklets and computer disks to DESE Industrial 
Education staff and three copies of the completed test-item booklets and computer disks to the 
Division of Vocational and Adult Education, Vocational Planning and Evaluation. 

12. Provide 53 booklets and computer disks to DESE Industrial Education staff to train electronics 
instructor'^> in test-item use during an in-sen/ice meeting at the 1991 Missouri Vocational Association 
Summer Conference. 




Trade and Technical - Drafting 

The general objective of this proposal was to develop criterion-referenced test items for the Missouri 
drafting competency profile. The test items could be adopted for use in either the VAMS program or the 
COMPVIMS/Project Basic III program. Successful completion of this general objective depended upon the 
completion of the following specific objectives. 

1. Assign a coordinator for the project. 

2. Select drafting instructors and a test-construction expert to participate in the test-item development 
workshop. 

3. Identify existing material (i.e., test banks or references) that may be useful In developing the test 
items. 

4. Conduct the test-item development workshop. The three-day workshop would consist of the 
following persons: drafting instructors; a test-construction expert; and representatives of DESE 
Industrial Education Section. 

5. Prepare the test items for validation by the test-construction expert; submit items for validation. 

6. Incorporate changes suggested by the test-construction expert. 

7. Select five field test sites, each in a different geographic location within Missouri. 

8. Orientate the field testers via teleconference. Field test the items. 

9. Conduct a meeting of field testers to evaluate items. 

10. Revise items according to the results of the field test. 

11. Deiiver two copies of the completed test-item booklets and computer disks to DESE Industrial 
Education staff and three copies of the completed test-item booklets and computer disks to the 
Division of Vocational and Adult Education, Vocational Planning and Evaluation. 

12. Provide 33 booklets and computer disks to DESE industrial Education staff to train drafting 
instructors in test-item use during an in-service meeting at the 1991 Missouri Vocational Association 
Summer Conference. 

Trade and Technical - Auto Mechanics 

The general objective of this proposal was to develop criterion-referenced test items for the Missouri 
auto mechanics competency profile. The test items could be used in either the VAMS program or the 
COMPVIMS/Project Basic III program. Successful completion of this general objective depended upon the 
completion of the following specific objectives. 

1. Assign a coordinator for the project. 

2. Select auto mechanics instructors and a test-construction expert to participate in the test item 
development workshop. 
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3. Identify existing material (i.e. test banks or references) that may be useful in developing the test 
items. 

4. Conduct the test item development workshop. The three-day workshop would consist of the 
following persons: auto mechanics instructors; a test-construction expert; and representatives of the 
DESE Industrial Education Section. 

5. Prepare the test items for validation by the test-construction expert; submit the items to the test- 
construction expert for review. 

6. Incorporate changes suggested by the test-construction expert. 

7. Select five field test sites, each in a different geographic location within Missouri. 

8. Crientate the field testers; field test the items. 

9. Conduct a meeting of field testers to evaluate items. 

10. Revise items according to the results of the field test. 

11. Print copies of the test items to be disseminated at the 1991 Missouri Vocational Association 
Summer Conference. 

12. Deliver two copies of the completed te?^t-item booklets and computer disks to DESE Industrial 
Education staff and two copies of the completed test-item booklets and computer disks to the 
Division of Vocational and Adult Education, Vocational Planning and Evaluation. 

12. Train auto mechanics instructors in the use of the test item bank and disseminate the bank during 
an in-service meeting at the 1991 Missouri Vocational Association Summer Conference. 

POPULATION AND SAMPLE 

Our cost-recovery materials are developed for all areas of vocational education for use by secondary 
schools, vocational-technical schools, junior colleges, and other institutions of higher and special education. 
Our fieldtesting and advisory committee assistance is drawn from the personnel involved in these 
educational institutions and industry representatives. 

I 

CONCLUSIONS AND RECOMMENDATIONS 

After the vocational educational curriculum priorities were determined, curriculum materials were 
developed under the direction of selected PAVTE faculty members and DESE personnel with the assistance 
of IML staff. Committees of local teachers, administrators, and business and industry representatives from 
the state were involved in setting curriculum development priorities and in identifying content for the 
curriculum resources developed. All of the guides in this report have undergone this developmental process. 
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DISCLAIMER 

The following disclaimer was shown on all curriculum disseminated as a result of this project: 
The activity which is the subject of this report was supported in whole or in part by funds from the 
Department of Elementary and Secondary Education, Division of Vocational and Adult Education. However, 
the opinions expressed herein do not necessarily reflect the position or policies of the Missouri Department 
of Elementary and Secondary Education or the Division of Vocational and Adult Education, and no official 
endorsement should be inferred." 
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Appendix 1 



Machine Shop 
Criterion'Referenced Test (CRT) Workshop 

Chancellor N (lower level), Days Inn 



Mon.. Aug. 6, 1990 

8:30-9: 1 5 a.m. Welcome and Introductions 

Diane Davis and Phyllis Miller 
Instructional Materials Laboratory. MU 

Opening remarks and project overview 
iG6\th Moore, supervisor 
Industrial Education, DESE 

Introduction to test item development 
Charles Oviatt 
Educational consultant 

Lunch (Chancellor VI) 

Division of competencies among writers 
Review of existing test items for possible inclusion 
--IML guide 
- Apprentice Machinist 
"Other sources 



Tues., Auq> 7 



8:30-1 1 :30 a.m. 


Write test items 




11:30-12:30 


Lunch (Chancellor VI) 




12:30-5 p.m. 


Discuss field-test sites and meeting(s) 

Progress review and continuation of test item development 




Wed., Aua. 8 






8:30-11:30 a.m. 


Continuation of test item development 




11:30-12:30 


Lunch (Chancellor VI) 




12:30-4 p.m. 


Conclusion of test Item development 




4-5 


Wrap-up; expense voucher completion 





9:15-11:30 

11:30-12:30 
12:30-5 p.m. 



ERIC 
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Appendix 1 (cont.] 



Writing Test Items 



Presenter: 
Charles D. Oviatt, Ph.D. 



Instructional Materials Laboratory 
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AGENDA FOR ORIENTATION SESSION 



Appendix 1 (cont,) 
1 



1. Domains and levels of learning 

2. Linking competencies to item types 

Computer-scored Manually scored 

True-false Performance checklist 

Matching 
Multiple-choice 

3. Advantages and disadvantages of item types 

4. Writing items for VIMS competencies 

5. Practice/feedback 

WORKSHOP OBJECTIVE 



Upon completion of this workshop, participants will have used educational measurement 
principles to create test items and checklists for the competencies (tasks) of the Vocational 
Instructional Management System (VIMS). Participants will write at least four computer- 
scorable test items for each competency. 
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Appendix 4 (cont) 



ADVANTAGES /DISADVANTAGES OF ITEM TYPES 



ADVANTAGES DISADVANTAGES 



TRUE- FALSE 


cover '^ content 
easy to write 
easy to score 


emphasis on meroory 
dependence on absolute 

j udgments 
^ guessing factor 


MATCHING 


easy to write 
measures assocs. 
^ guessing 
factor 


emphasis on memory 
scanned answer sheets 
accommodate 5 opts. 


MULTIPLE-CHOICE 


versatile 
cover ^ content 
easy to score 
guessing 

factor 
useful for 

instruction 


time consuming 
to write, esp. 
for high levels 


COMPLETION 


easy to write 

^ guessing factor 

cover ^ content 


scoring can be 
difficult 

emphasis on memory 


RESTRICTED-RESPONSE 
ESSAY 


easy to write 
measures expression 

St organization 
nJ/ guessing factor 


time consuming to 

score 
subjective scoring 
cover "i/ content 


CHECKLIST 


meaauras actual 
performance 


subjective scoring 
time consximing to 



d«v«lop and 

adininistar 
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THE TRUE-FALSE ITEM 



The true-false (alternative response) item consists of a 
declarative statement that the examinee indicates is either 
true or false (right or wrong, correct or incorrect, etc-)- 

The true-false item is most commonly used to measure the 
ability to identify the correctness of facts, definitions of 
terms, statements of principles and the like- It can also 
be used to distinguish cause from effect or correct from 
incorrect procedures- 

Because the true-false item has several limitations, 
especially that of being susceptible to guessing, it should 
be used sparingly. 

Rules for Writing True-False Items 



DO: 1. explain method of marking (+/0, T/F, etc-) 

2- construct statements that are definitely true or 
false 

3. attribute an opinion to its source 

4. write short state;nents 

5- keep true and false statements equal in length 

6- include more false items than true items (maybe 
2/3 false; 1/3 true) 



DON'T: 1- write double negative statements 

2. give verbal clues (such as absolutes like 
"always" or "never") 

3- order items according to some system that will 
give clues. 

4. take statements directly from text 

5. write complex statements with two complete parts 

6- write general statements which need 
qualifications (or more information) 

7- write items using second person C'you") 
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THE MATCHING ITEM 

Matching items consist of a set or list of descriptions 
(stimuli) and a set of options (responses) which are to be 
matched or associated. 

Matching items are commonly used to measure factual 
information based on simple associations - for example, 
symbols and concepts or objects and labels. 

The matching item can be used to measure objectives which 
emphasize the ability to identify the relationship between 
two things as long as a sufficient number of descriptions 
and options can be obtained. 

Rules for Writing Matching Items 

DO: 1. write clear directions and indicate if options 

can be used more than once 

2. keep the list of descriptions and the list of 
options short and homogeneous 

3. title the lists and arrange in logical order 

4. make sure all options are plausible 

5. make descriptions the longer phrases; options 
shorter 

6. number each description; letter each option 

7. include more options than descriptions and 
include some options that don't match any 
descriptions 



DON*T: 1. put any part of either list on different pages 
or let descriptions be separated from options. 

2. exceed 7 or 8 descriptions 
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THE MULTIPLE-CHOICE ITEM 



A multiple-choice item consists of a problem and a list of 
suggested solutions. The problem may be stated as a^ direct 
question or as an incomplete statement. This is called the 
stem. The list of suggested solutions may include words ^ 
numbers, phrases, symbols, etc. which are called options, 
choices, or alternatives. The correct option is the answer 
and the remaining options are called distractors (or decoys 
or foils ) . 

The multiple-choice item is the most versatile type of test 
item available. It can measure a variety of learning 
outcomes from simple to complex, and it is adaptable to any 
content area. It is typically used to measure knowledge of 
terminology, specific facts, principles, methods, 
procedures • and other types of higher order cognitive 
skills. 



Rules for Writing Multiple-Choice Items 

DO: !• formulate a clear problem in the stem 

2. provide only one correct answer 

3, make sure distractors are plausible 

4, eliminate grammatical clues in stem (''a" or "an*') 

5, keep distractors of 'equal length 

6. rotate position of correct answer 

7 . include 3 to 5 options 

8, make options similar in terms of content 

9. order options according to a logical system 

DON'T: !• use negatively-stated questions or statements 
excessively 

2, use "none of the above" frequently 

3, use "all of the above" frequently 

4, use "a and b" , "b, c, and d" etc, at all 

5, use overlapping alternatives 

6, repeat information/words occurring in stem in options 
ERJj^ 7* write items using second person ("you") 



THE PERFORiMANCE CHECKLIST 



Performance checklists contain a list of behaviors^ traitirs, 
or characteristics that are either present or absent. 
Objectives that require a student to demonstrate psychomoto 
skills (e.g., procedures) can be effectively measured by 
performance checklists . 

Constructing checklists requires a thorough analysis of the 
behaviors required to perform a procedure satisfactorily. 
Steps in constructing a VIMS checklist for a psychomotor 
performance objective include: 

1. identifying the behavioral segments of the behavior 
(TASK) specified in the objective, 

2. sequencing those behavioral segments correctly, and 

3- putting the behavior analysis in rhe form of a 
checklist . 



STEPS IN WRITING TEST ITEMS/CHECKLISTS FOR VIMS COMPETENCIE 

1. Determine the domain and le- el of learning implied by 
the competency. 

2. Select the appropriate item type with which to measure 
the competency, given the domain and level of learning. 

3. Determine the content parameters specified in the 
competency . 

4. Write at least four congruent computer-scorable items 
and, when appropriate, one congruent checklist for each 
competency . 



Remember: Congruence between the item and the 
competency is a function of level alignment and 
content match! 



